Comparing Supervised Machine Learning Classification Methods to Identify Risk Factors for Suicide Morbidity Among USA High School Students

Catalina Cañizares. MSc; Mark J. Macgowan. Ph.D.; Gabriel Odom. Ph.D., Th.D.

Agenda

Definitions
Background Suicide Morbidity
Significance of the study
Research aims
Theoretical Framework
- Suicide Morbidity
- Machine Learning
Methods
Ethical Implications

WHO defines Adolescents as individuals in the 10-19 years age (WHO, n.d.)
Is the phase of life between childhood and adulthood (WHO, n.d.)
Adolescents experience rapid physical, cognitive and psychosocial growth (WHO, n.d.)

Any self-reported thoughts of engaging in suicide-related behaviors (O’Carroll et al. 1996)
- Considering
- Planning suicide

Any act that is self-inflicted and potentially injurious, for which there is evidence of intent to die (Silverman et al. 2007)
A suicide attempt may result in death, injury, or no injury.
Suicide attempt I: No injury
Suicide attempt II: Any degree of injury (Silverman et al. 2007)

Background

Facts
Trends

Suicide is the third leading cause of death among 15-19 year-olds (CDC 2020b).
One in five (18.8%) students nationwide reported suicide ideation (CDC 2020a).
One in six (15.7%) students has made a suicide plan (CDC 2020a)
One in 11 (8.9%) has attempted suicide at least one time in their lifetime (CDC 2020a)
Suicide ideation and suicide attempts are the most commonly reported mental health crises among youth (Standley 2020)

Code

library(tidyYRBS)
library(geomtextpath)
library(tidyverse)

data("hs_suicide")
data("hs_demographics")

the_data <- left_join(hs_demographics, hs_suicide)

# Weights
the_data_weights <- the_data |>
  srvyr::as_survey_design(
    ids=PSU,
    weights=weight,
    strata=stratum,
    nest = TRUE
  )

# Preparing the data for the ggplot
considered <- the_data_weights %>% 
  group_by(year) %>% 
  summarise(prevalence = mean(suicide_considered, na.rm = TRUE),
            n = n()) %>% 
  mutate(origin = "Considered")

attempts <- the_data_weights %>%
  mutate(
    suicide_attempts = case_when(
      suicide_attempts == 0 ~ FALSE,
      suicide_attempts %in% 1:6 ~ TRUE,
      TRUE ~ NA
    )
  ) %>%
  group_by(year) %>%
  summarise(
    prevalence = mean(suicide_attempts, na.rm = TRUE),
    n = n()
  ) %>%
  mutate(origin = "Attempts")
  
complete_data <- considered %>% 
  rbind(attempts)

ggplot(complete_data , aes(year, prevalence, label = origin, color = origin)) +
  geom_smooth(alpha = 0.1, size = 0) +
  geom_textline(hjust = .40, size = 10) +
  scale_color_manual(values = c("#4e2d86", "#24bccb")) + 
  theme_minimal(base_size = 28) +
  theme(legend.position = "none") +
  theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank()) +
  scale_x_continuous(breaks = seq(1990, 2020, 2)) +
  theme(axis.text.x = element_text(angle = 90)) +
  theme(panel.background = element_rect(fill = "#f5fafc",
                                colour = "#f5fafc")) +
  theme(plot.background = element_rect(fill = "#f5fafc", colour = "#f5fafc")) +
  scale_y_continuous(lim=c(.0, .35),
                     breaks = seq(0, 1, 0.05),
                     labels = scales::percent) +
  labs(y="Suicide Morbidity Prevalence", x="",
       title="Youth Prevalence of Suicide Morbidity",
       caption = "Data from: YRBS, 1990-2019, tidyYRBS")

Past studies

Evaluated 66 studies from 2015 to 2019
Internal risk factors:
- Ineffective coping
- Poor lifestyle
- Disturbed sleep
External risk factors
- Family history of mental health
- Poor interactions in the family

Evaluated 67 population-based longitudinal studies
- A history of previous suicidal thoughts and behaviors
- Family history of mental disorders
- Physical and psychological abuse

Evaluated 365 longitudinal studies of the past 50 years of research
Risk factors have been homogeneous over time
- Demographic characteristics
- Internalizing psychopathology
- Prior history of suicide attempts
- Externalizing psychopathology
- Social factors

Gaps

The ability to predict suicide morbidity has been near chance for past 50 years of research (Franklin et al. 2017)

Most findings come from research designs that explore the effect of single risk factors (Franklin et al. 2017; Burke et al. 2018; Burke, Ammerman, and Jacobucci 2019; Ribeiro et al. 2016)
Although some models have used more than a single risk factor, most research relies on traditional statistical approaches that restrict the number of variables that can be simultaneously examined, creating overly simplistic models (Franklin et al. 2017)
Traditional models presuppose that the researcher must define the relationship between predictors and outcomes a priori (Burke, Ammerman, and Jacobucci 2019; Cox et al. 2020; Linthicum, Musacchio, and Ribeiro 2019).
Theoretically, the processes that facilitate suicide morbidity are complex and entail multiple interactions; therefore, any risk factor considered in isolation will be an inaccurate predictor

A shift in research is needed to capture the complexities behind adolescent suicide morbidity

Significance of the study

Study suicide morbidity as a complex classification problem that considers multiple risk factors simultaneously (Ribeiro et al. 2016; Linthicum, Musacchio, and Ribeiro 2019).
Machine Learning methods considers highly complex relations among variables to determine the optimal classification algorithm (Ribeiro et al. 2016; Bernert et al. 2020; Linthicum, Musacchio, and Ribeiro 2019).

Study suicide morbidity in adolescents:

Using flexible methodological techniques
Using methods with better predictability performance
Risk algorithms instead of single risk factors

Machine learning in Suicidology

35 independent studies used ML to predict suicide-related events
More accurate levels of performance in predictions over traditional statistical methodology (AUCs = 0.80–0.84)

There is a scarcity of research using adolescent population

Research aims

Identify the critical risk factors for adolescent suicide morbidity from a set of 99 risk behavior predictors with machine learning classification algorithms.
Identify the best machine learning methodology to classify adolescents who attempted and considered suicide according to its classification performance (ROC, overall accuracy, and the Kappa value).
Compare the performance of an a priori-determined model to models informed by feature selection from the least absolute shrinkage and selection operator method.
Identify if there are differences in the critical risk factors for suicide ideation and suicide attempts.

Socioecological model for suicide morbidity

Applied to youth
Applied to this research
Rationale

Conceives human development as the constant interaction between the individual and the changing environment in which it lives and grows (Bronfenbrenner 1977).

Ontogenic

Sex
Race
Age

Microsystem

Family members
Friends
School

Exosystem

The media
Neighborhood

Macrosystem

Economic, social, educational, legal, and political systems

Allows to study adolescent suicide morbidity as the interaction of multiple risk factors at multiple levels of the adolescent system (Perkins and Hartless 2002).
Moves beyond the tendency to evaluate only individualistic characteristics of adolescents.
Allows the assessment of other factors that have proved relevant in the likelihood of suicide attempts and ideation (Ayyash-Abdo 2002; Price and Khubchandani 2017)

Machine Learning

Supervised Machine Learning
Simple example
Rationale

Supervised machine learning learns from data to detect patterns (Elhai and Montag 2020; Teboul 2018)

Will find the function that maps the predictors to the outcomes (Stewart 2020)
The result is an algorithm representing the closest possible match to the behavior of the data
Before the algorithm is tested, model tuning is performed to achieve more accurate predictions
The tuning is performed for the hyperparameters (Kuhn and Silge 2022).
The algorithm is evaluated by its performance in predicting the outcome (Kuhn and Silge 2022)
Receiver operating characteristic (ROC), overall accuracy, and the Kappa value

How supervised learning works. Source: Pickell, Devin. 2021. “Supervised Vs Unsupervised Learning – What’s the Difference?” G2. https://www.g2.com/articles/supervised-vs-unsupervised-learning.

ML methods are recommended and well suited for classification problems that involve high dimensional data (a large number of potential predictors) (Iniesta, Stahl, and McGuffin 2016; Walsh, Ribeiro, and Franklin 2018; Elhai and Montag 2020; Bzdok, Altman, and Krzywinski 2018)
ML is resistant to overfitting high-dimensional data because it integrates resampling strategies (Kuhn and Silge 2022; Walsh, Ribeiro, and Franklin 2018)
ML models focus on finding the predictive pattern in the data and how accurate this pattern is when classifying new cases (Bzdok, Altman, and Krzywinski 2018; Walsh, Ribeiro, and Franklin 2018)
ML has the advantage of taking into account all the available information (Rajula et al. 2020)

Methods

The Data
Participants
Measures
Data analysis

Youth Risk Behavior Surveillance System (YRBSS)
Combined YRBS High School Dataset
Surveys that monitors health behaviors and experiences among high school students in grades 9–12 attending U.S. public and private schools since 1991 (Underwood et al. 2020)

The main categories included in the surveys:

Behaviors that contribute to unintentional injury and violence
Tobacco use
Alcohol and other drug use
Sexual behaviors that contribute to unintended pregnancy and STD/HIV infection
Dietary behaviors
Physical inactivity

tidyYRBS

Code

# Libraries
library(srvyr)
library(scales)

# Loading the complete dataset
data("hs_district")

# The number of participants unweighted
n_yrbs <- 
  nrow(hs_district) %>% 
  comma()

saveRDS(n_yrbs, "data/n_yrbs.rds")


# This function transforms the Data Frame into a survey object
yrbs_df <-
  hs_district %>%
  srvyr::as_survey_design(
    ids     = PSU,
    weights = weight,
    strata  = stratum,
    nest    = TRUE
  )

# N weighted
total_weight <- 
  yrbs_df %>% 
  summarise(N = survey_total()) %>% 
  select(N) %>% 
  pull() %>% 
  comma()

saveRDS(total_weight, "data/total_weight.rds")

# Sex weighted

female <- 
  yrbs_df %>% 
  group_by(sex) %>%
  summarise(N = survey_total()) %>% 
  mutate(sex = as.character(haven::as_factor(sex))) %>% 
  filter(sex == "Female") %>% 
  select(N) %>% 
  pull() %>% 
  comma()

saveRDS(female, "data/female.rds")

male <- 
  yrbs_df %>% 
  group_by(sex) %>%
  summarise(N = survey_total()) %>% 
  mutate(sex = as.character(haven::as_factor(sex))) %>% 
  filter(sex == "Male") %>% 
  select(N) %>% 
  pull() %>% 
  comma()

saveRDS(male, "data/male.rds")

# Suicide attempts
data("hs_suicide")

suicide_df <- 
  hs_district %>% 
  dplyr::select(weight, stratum, PSU, record) %>% 
  mutate(record = as.character(record)) %>% 
  left_join(hs_suicide) %>% 
  mutate(
    suicide_attempts = case_when(
                    suicide_attempts == 0 ~ FALSE, 
                    suicide_attempts %in% 1:6 ~ TRUE, 
                    TRUE ~ NA)
  )


suicide_data <-
  suicide_df %>%
  srvyr::as_survey_design(
    ids     = PSU,
    weights = weight,
    strata  = stratum,
    nest    = TRUE
  )

suicide_attempts_df <- 
  suicide_data %>% 
  group_by(suicide_attempts) %>%
  summarise(proportion = survey_mean(),
            total = survey_total()) %>% 
  dplyr::filter(suicide_attempts == TRUE) %>% 
  pull(proportion) %>% 
  scales::percent()

saveRDS(suicide_attempts_df, "data/suicide_attempts.rds")

# Suicide ideation

suicide_considered_df <- 
  suicide_data %>% 
  group_by(suicide_considered) %>%
  summarise(proportion = survey_mean(),
            total = survey_total()) %>% 
  dplyr::filter(suicide_considered == TRUE) %>% 
  pull(proportion) %>% 
  scales::percent()

saveRDS(suicide_considered_df, "data/suicide_considered.RDS")

The total weighted sample for the Combined YRBS High School Dataset is 14,395,146 cases.
From these, 7,159,104 are female, and 7,141,727 are male.
The proportion of students that reported attempting suicide in this data is 8%
The proportion of students that considered suicide is 15%.

Outcomes:

(Q26) During the past 12 months, did you ever seriously consider attempting suicide?
(Q28) During the past 12 months, how many times did you actually attempt suicide?

Predictors:

Demographic variables (age, sex, grade, race, sexual identity, site, year)

Questionnaire items (q8-q99)

Logistic Regression, Lasso, K-Nearest Neighbors, Random Forest, Classification and Regression Trees, and Extreme Gradient will be used to generate the predictive models
To create the models, the tidymodels (Kuhn and Wickham 2020)
The complete dataset will be divided into two datasets: 75% for training 25% for testing (Kuhn and Silge 2022).
The testing dataset will be set to make 10-fold cross-validation to tune by the relevant hyperparameters for each technique
The best model will be selected according to the highest value of ROC-AUC, overall accuracy, and Kappa value (Kuhn and Silge 2022).

Machine Learning Methods

Logistic Regression
Lasso regression
K-Nearest Neighbors (KNN)
Classification Trees
Random Forest
Extreme Gradient Boosting (XGBoost)

Model the outcome as a linear function of the predictors (Burkov 2019).
The sigmoid function is applied to adjust the predictions to stay between 0 and 1 (Burkov 2019)
The predictors will be selected from past literature modeling YRBSS data (Bae et al. 2005)

Logistic regression gif Source: Laken, Paul van der. 2020. “Animated Machine Learning Classifiers.” Paulvanderlaken.com. https://paulvanderlaken.com/2020/01/20/animated-machine-learning-classifiers/.

Estimates the coefficients aiming at zero, this means that it uses shrinkage (James et al. 2013).
It is extremely useful for variable importance, selection, and regularization (James et al. 2013).
This technique will select only relevant coefficients (James et al. 2013).

KNN will assign class membership to a data point with the majority vote of its K-nearest neighbors.

Divides the feature space into non-overlapping rectangular regions with similar response rates that can later be used for prediction (Greenwell 2022).

Random forest consists of hundreds or thousands of independently grown decision trees generated from different bootstrap samples from the training data (Greenwell 2022).
Uses hundreds of trees in the back end and thus results in a more flexible boundary

Same concept of Random Forest but..
Each additional tree added to the model partially fixes the errors made by the previous trees until the maximum number of trees are combined (Burkov 2019)

Ethical Implications

Minimal human subjects concerns

The original data was not collected to answer the present research question
Acknowledge the ownership of the original data.
Accidentally losing or destroying the data.

To wrap up

Just in case…

Cross-validation
Evluation coefficientes
Confussion matix
Predictors in Logistic

Accuracy: is the fraction of predictions our model got right.

ROC: A receiver operating characteristic curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier. It is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.

Kappa: how closely the instances classified by the machine learning classifier matched the data labeled as ground truth

Thank you

References

Ayyash-Abdo, Huda. 2002. “Adolescent Suicide: An Ecological Approach.” Psychology in the Schools 39 (July): 459–75. https://doi.org/10.1002/pits.10042.

Bae, S., R. Ye, S. Chen, P. A. Rivers, and K. P. Singh. 2005. “Risky Behaviors and Factors Associated with Suicide Attempt in Adolescents.” Journal Article. Arch Suicide Res 9 (2): 193–202. https://doi.org/10.1080/13811110590904034.

Bernert, R. A., A. M. Hilberg, R. Melia, J. P. Kim, N. H. Shah, and F. Abnousi. 2020. “Artificial Intelligence and Suicide Prevention: A Systematic Review of Machine Learning Investigations.” Journal Article. Int J Environ Res Public Health 17 (16). https://doi.org/10.3390/ijerph17165929.

Bronfenbrenner, Urie. 1977. “Toward an Experimental Ecology of Human Development.” American Psychologist 32: 513–31.

Burke, Taylor A., Brooke A. Ammerman, and Ross Jacobucci. 2019. “The Use of Machine Learning in the Study of Suicidal and Non-Suicidal Self-Injurious Thoughts and Behaviors: A Systematic Review.” Journal of Affective Disorders 245 (February): 869–84. https://doi.org/10.1016/j.jad.2018.11.073.

Burke, Taylor A., Ross Jacobucci, Brooke A. Ammerman, Marilyn Piccirillo, Michael S. McCloskey, Richard G. Heimberg, and Lauren B. Alloy. 2018. “Identifying the Relative Importance of Non-Suicidal Self-Injury Features in Classifying Suicidal Ideation, Plans, and Behavior Using Exploratory Data Mining.” Psychiatry Research 262 (April): 175–83. https://doi.org/10.1016/j.psychres.2018.01.045.

Burkov, A. 2019. The Hundred-Page Machine Learning Book. Andriy Burkov. https://books.google.com/books?id=ZF3KwQEACAAJ.

Bzdok, Danilo, Naomi Altman, and Martin Krzywinski. 2018. “Statistics Versus Machine Learning.” Journal Article. Nature Methods 15 (4): 233–34. https://doi.org/10.1038/nmeth.4642.

CDC. 2020a. “2019 YRBS National, State, and District Combined Datasets User’s Guide.” Dataset.

———. 2020b. “America’s Health Rankings Analysis of CDC WONDER, Multiple Cause of Death Files, United Health Foundation.” Web Page. https://www.americashealthrankings.org/explore/health-of-women-and-children/measure/teen_suicide/state/ALL.

Cox, Christopher R., Emma H. Moscardini, Alex S. Cohen, and Raymond P. Tucker. 2020. “Machine Learning for Suicidology: A Practical Review of Exploratory and Hypothesis-Driven Approaches.” Clinical Psychology Review 82 (December): 101940. https://doi.org/10.1016/j.cpr.2020.101940.

Elhai, Jon D, and Christian Montag. 2020. “The Compatibility of Theoretical Frameworks with Machine Learning Analyses in Psychological Research.” Current Opinion in Psychology 36: 83–88. https://doi.org/https://doi.org/10.1016/j.copsyc.2020.05.002.

Franklin, Joseph C., Jessica D. Ribeiro, Kathryn R. Fox, Kate H. Bentley, Evan M. Kleiman, Xieyining Huang, Katherine M. Musacchio, Adam C. Jaroszewski, Bernard P. Chang, and Matthew K. Nock. 2017. “Risk Factors for Suicidal Thoughts and Behaviors: A Meta-Analysis of 50 Years of Research.” Psychological Bulletin 143 (2): 187–232. https://doi.org/10.1037/bul0000084.

Greenwell, Brandon M. 2022. Tree-Based Methods for Statistical Learning in r. Chapman & Hall/CRC Data Science Series. Boca Raton, FL: CRC Press.

Iniesta, R., D. Stahl, and P. McGuffin. 2016. “Machine Learning, Statistical Learning and the Future of Biological Research in Psychiatry.” Psychological Medicine 46 (12): 2455–65. https://doi.org/10.1017/S0033291716001367.

James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. An Introduction to Statistical Learning: With Applications in r. Springer. https://faculty.marshall.usc.edu/gareth-james/ISL/.

Kuhn, Max, and Julia Silge. 2022. Tidy Modeling with r. Book. O’Reilly Media, Inc.

Kuhn, Max, and Hadley Wickham. 2020. Tidymodels: A Collection of Packages for Modeling and Machine Learning Using Tidyverse Principles. https://www.tidymodels.org.

Linthicum, Kathryn, Katherine Musacchio, and Jessica Ribeiro. 2019. “Machine Learning in Suicide Science: Applications and Ethics.” Behavioral Sciences & the Law 37 (May). https://doi.org/10.1002/bsl.2392.

O’Carroll, P, A Berman, R W Maris, E K Moscicki, B L Tanney, and M M Silverman. 1996. “Beyond the Tower of Babel: A Nomenclature for Suicidology.” Journal Article. Suicide Life Threat Behav 26 (3): 237–52.

Perkins, Daniel, and Glen Hartless. 2002. “An Ecological Risk-Factor Examination of Suicide Ideation and Behavior of Adolescents.” Journal of Adolescent Research - J ADOLESCENT RES 17 (January): 3–26. https://doi.org/10.1177/0743558402171001.

Price, James, and Jagdish Khubchandani. 2017. “Latina Adolescents Health Risk Behaviors and Suicidal Ideation and Suicide Attempts: Results from the National Youth Risk Behavior Survey 2001–2013.” Journal of Immigrant and Minority Health 19 (June): 533–42. https://doi.org/10.1007/s10903-016-0445-8.

Rajula, H. S. R., G. Verlato, M. Manchia, N. Antonucci, and V. Fanos. 2020. “Comparison of Conventional Statistical Methods with Machine Learning in Medicine: Diagnosis, Drug Development, and Treatment.” Journal Article. Medicina (Kaunas) 56 (9). https://doi.org/10.3390/medicina56090455.

Ribeiro, J. D., J. C. Franklin, K. R. Fox, K. H. Bentley, E. M. Kleiman, B. P. Chang, and M. K. Nock. 2016. “Letter to the Editor: Suicide as a Complex Classification Problem: Machine Learning and Related Techniques Can Advance Suicide Prediction - a Reply to Roaldset (2016).” Journal Article. Psychol Med 46 (9): 2009–10. https://doi.org/10.1017/s0033291716000611.

Silverman, Morton M., Alan L. Berman, Nels D. Sanddal, W. O’Carroll Patrick, and Joiner Thomas E. 2007. “Rebuilding the Tower of Babel: A Revised Nomenclature for the Study of Suicide and Suicidal Behaviors: Part 2: Suicide-Related Ideations, Communications, and Behaviors.” Suicide & Life - Threatening Behavior 37 (3): 264–77. https://www.proquest.com/scholarly-journals/rebuilding-tower-babel-revised-nomenclature-study/docview/224871202/se-2.

Standley, Corbin. 2020. “Expanding Our Paradigms: Intersectional and Socioecological Approaches to Suicide Prevention.” Death Studies 46 (February). https://doi.org/10.1080/07481187.2020.1725934.

Stewart, Matthew. 2020. “The Actual Difference Between Statistics and Machine Learning.” Medium. Towards Data Science. https://towardsdatascience.com/the-actual-difference-between-statistics-and-machine-learning-64b49f07ea3.

Teboul, Wendy. 2018. “Why Use Machine Learning Instead of Traditional Statistics?” Medium. Towards Data Science. https://towardsdatascience.com/why-use-machine-learning-instead-of-traditional-statistics-334c2213700a.

Underwood, Michael, Nancy Brener, Jemekia Thornton, William A. Harris, Leah N. Bryan, Shari L. Shanklin, Nicholas Deputy, et al. 2020. “Overview and Methods for the Youth Risk Behavior Surveillance System — United States, 2019.” Journal Article. MMWR, 1–10. https://doi.org/ http://dx.doi.org/10.15585/mmwr.su6901a1external i.

Walsh, Colin G., Jessica D. Ribeiro, and Joseph C. Franklin. 2018. “Predicting Suicide Attempts in Adolescents with Longitudinal Clinical Data and Machine Learning.” Journal of Child Psychology and Psychiatry 59 (12): 1261–70. https://doi.org/10.1111/jcpp.12916.

WHO. n.d. “Adolescent Health.” Web Page. https://www.who.int/southeastasia/health-topics/adolescent-health#:~:text=WHO%20defines%20'Adolescents'%20as%20individuals,East%20Asia%20Region%20(SEAR).